Image and Audio Annotation: Approximate Inference in Dense Conditional Random Fields

In the task of image and audio annotation, it is typical for labels to be assigned independently. While some correlations may be modeled, it is rare for all interdependencies between labels to be considered. This is because exact inference on a dense, cyclical graphical model is often intractable. This paper applies an approximate inference method, log-determinant relaxation (LDR), to a fully connected conditional random field (CRF) to perform automatic image and audio annotation. The proposed model estimates the conditional distribution of all labels in a vocabulary given the audio or image features. Two other discriminative models incorporating a varying degree of context are used as a baseline for comparison. We show that introducing context to CRFs improves annotation performance, and can be made tractable with LDR. At the time of writing, the application of LDR to a discriminative model is novel.